Storage system and control software deployment method

ABSTRACT

A storage system includes a plurality of storage nodes. The storage node includes: one or more storage devices which respectively provide a storage area; and one or more control software which read/write requested data from/into the corresponding storage device according to a request from a higher-level device, wherein each of the control software retains predetermined configuration information required for reading/writing requested data from/into the corresponding storage device according to a request from the higher-level device, wherein a plurality of the control software are managed as a redundancy group, and the configuration information retained in each of the control software belonging to the same redundancy group is synchronously updated, and wherein the plurality of control software configuring the redundancy group are each deployed in respectively different storage nodes so as to distribute a load of each of the storage nodes.

TECHNICAL FIELD

The present invention relates to a storage system and a control softwaredeployment method and, for instance, can be suitably applied to aninformation processing system comprising a plurality of storage nodeseach equipped with one or more SDSs (Software Defined Storages). Notethat, in the ensuing explanation, the term “SDS” refers to a storageapparatus that is configured by loading software, which has a storagefunction, in a general purpose server device.

BACKGROUND ART

Conventionally, with an information processing system, a redundancyconfiguration of the server device is often adopted for improvingavailability and reliability. Thus, upon improving the processingperformance of the overall system in a conventional informationprocessing system, the scale-out method of adding/reducing physicalservers in units of a plurality of server devices configuring theredundancy configuration is being broadly adopted.

Nevertheless, according to this kind of conventional scale-out method, aplurality of physical servers need to be added for redundancy even whenadding the server function of only one server, and there was a problemin that costs would increase by that much.

Meanwhile, in recent years, the virtualization technology of running aplurality of virtual machines on a single physical server is becomewidely prevalent, and numerous inventions have been proposed for theredundancy of virtual machines. For example, PTL 1 discloses atechnology of arranging, in a plurality of physical servers, a pluralityof virtual machines of active systems and virtual machines of reservesystems each provided in correspondence with the foregoing virtualmachines of active systems.

According to this kind of virtualization technology, even in cases ofadding a group of active system virtual machines and their reservesystem virtual machines (this is hereinafter referred to as the “virtualmachine group”), there is an advantage in that it is not necessary toadd the same number of physical servers as the number of virtualmachines configuring the virtual machine group, and scale-out can beperformed efficiently and inexpensively.

CITATION LIST Patent Literature

-   PTL 1: Japanese Unexamined Patent Application Publication No.    2014-75027

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

PTL 1 describes a technology of respectively arranging the active systemvirtual machines and the reserve system virtual machines in separatephysical servers (active system-dedicated physical server or reservesystem-dedicated physical server). Here, for instance, by arrangingcertain active system virtual machines and certain reserve systemvirtual machines in the same physical server, it may be possible toeffectively utilize information processing resources as an overallsystem.

Thus, proposed is a storage control software deployment method capableof effectively utilizing information processing resources as an overallsystem while maintaining the availability and reliability required in astorage system.

Means to Solve the Problems

In order to achieve the foregoing object, the present invention providesa storage system including a plurality of storage nodes, wherein thestorage node comprises: one or more storage devices which respectivelyprovide a storage area; and one or more control software whichread/write requested data from/into the corresponding storage deviceaccording to a request from a higher-level device, wherein each of thecontrol software retains predetermined configuration informationrequired for reading/writing requested data from/into the correspondingstorage device according to a request from the higher-level device,wherein a plurality of the control software are managed as a redundancygroup, and the configuration information retained in each of the controlsoftware belonging to the same redundancy group is synchronouslyupdated, and wherein the plurality of control software configuring theredundancy group are each deployed in respectively different storagenodes so as to distribute a load of each of the storage nodes.

Moreover, the present invention additionally provides a control softwaredeployment method of deploying, in a storage system including aplurality of storage nodes each equipped with one or more storagedevices, one or more control software which read/write requested datafrom/into the corresponding storage device according to a request from ahigher-level device, wherein each of the control software retainspredetermined configuration information required for reading/writingrequested data from/into the corresponding storage device according to arequest from the higher-level device, wherein a plurality of the controlsoftware are managed as a redundancy group, and wherein control softwaredeployment method comprises: a first step of deploying each of theplurality of control software configuring the redundancy group inrespectively different storage nodes so as to distribute a load of eachof the storage nodes; and a second step of synchronously updating theconfiguration information retained in each of the control softwarebelonging to the same redundancy group.

Advantageous Effects of the Invention

According to the present invention, it is possible to realize a storagesystem and a control software deployment method capable of effectivelyutilizing information processing resources as an overall system. Otherobjects, configurations and effects will become apparent based on theexplanation of the ensuing embodiments.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram showing the overall configuration of theinformation processing system according to this embodiment.

FIG. 2 is a block diagram showing the schematic configuration of thestorage node.

FIG. 3 is a block diagram explaining the programs and information storedin the memory of the storage node in relation to the redundancyfunction.

FIG. 4 is a chart showing a configuration example of thelogical-physical conversion table.

FIG. 5 is a chart showing a configuration example of the storage nodemanagement table.

FIG. 6 is a chart showing a configuration example of the controlsoftware management table.

FIG. 7 is a block diagram explaining the redundancy function accordingto this embodiment.

FIG. 8 is a block diagram explaining the redundancy function accordingto this embodiment.

FIG. 9 is a block diagram explaining the redundancy function accordingto this embodiment.

FIG. 10 is a block diagram explaining the redundancy function accordingto this embodiment.

FIG. 11 is a block diagram explaining the redundancy function accordingto this embodiment.

FIG. 12 is a block diagram explaining the redundancy function accordingto this embodiment.

FIG. 13 is a block diagram explaining the redundancy function accordingto this embodiment.

FIG. 14 is a flowchart showing the processing routine of the redundancyrecovery processing.

FIG. 15 is a flowchart showing the processing routine of the deploymentdestination storage node determination processing.

FIG. 16 is a flowchart showing the processing routine of the rebalanceprocessing.

FIG. 17 is a flowchart showing the processing routine of the migrationtarget control software determination processing.

DESCRIPTION OF EMBODIMENTS

An embodiment of the present invention is now explained in detail withreference to the appended drawings. The following descriptions anddrawings are illustrated for explaining the present invention, andcertain contents have been omitted or simplified as needed in order toclarify the explanation. Moreover, all combinations of the featuresexplained in the embodiments may not be required for achieving thesolution of the invention. Without limitation to the embodiments of thepresent invention, any and all applications that coincide with theconcept of the present invention are covered by the technical concept ofthe present invention. The present invention may be subject to variousadditions or changes within the scope of the present invention by thoseskilled in the art. The present invention can also be worked based onvarious other modes. Unless specifically limited, the respectiveconstituent elements may be plural or singular.

In the ensuing explanation, while various types of information will beexplained using expressions such as “table”, “chart”, “list”, and“queue”, the various types of information may also be expressed as otherdata structures. In order to indicate that certain information is notdependent on a data structure, “XX table”, “XX list” and the like arereferred to as “XX information”. Expressions such as “identificationinformation”, “identifier”, “name”, “ID”, and “number” are used in orderto explain the contents of each piece of information, and theseexpressions may be mutually substituted.

Moreover, in the ensuing explanation, reference characters or commonnumbers in such reference characters are used when explaining similarelements without differentiation, and a reference character of aspecific element is used, or an ID assigned to that element is used insubstitute for such reference character, when differentiating andexplaining similar elements.

Moreover, in the ensuing explanation, processing that is performed byexecuting programs will be explained. Here, because a program performspredetermined processing by using a storage resource (for example,memory) and/or an interface device (for example, communication port) asneeded by being executed by at least one or more processors (forexample, CPUs), the subject of processing may also be indicated as beingthe processor. Similarly, the subject of processing that is performed byexecuting programs may also be a controller, a device, a system, acomputer, a node, a storage system, a storage apparatus, a server, amanagement computer, a client, or a host equipped with a processor. Thesubject of processing (for example, processor) that is performed byexecuting programs may also include a hardware circuit which performs apart or all of the processing. For example, the subject of processingthat is performed by executing programs may include a hardware circuitwhich executes encryption and decryption, or compression anddecompression. The processor operates as a function part which realizespredetermined functions by being operated according to programs. Adevice and a system including a processor are a device and a systemincluding such function parts.

Programs may also be installed from a program source into a device suchas a computer. The program source may be, for instance, a programdistribution server or a computer-readable storage media. When theprogram source is a program distribution server, the programdistribution server includes a processor (for example, CPU) and astorage resource, and the storage resource may additionally store adistribution program and programs to be distributed. Furthermore, as aresult of the processor of the program distribution server executing thedistribution program, the processor of the program distribution servermay distribute the programs to be distributed to other computers.Moreover, in the ensuing explanation, two or more programs may berealized as one program, and one program may be realized as two or moreprograms.

(1) Configuration of Information Processing System According to thisEmbodiment

FIG. 1 is a diagram showing the configuration of the informationprocessing system 1 according to this embodiment. The informationprocessing system 1 is configured by comprising a plurality of computenodes 2, and a plurality of storage nodes 3.

The respective compute nodes 2 and the respective storage nodes 3 areconnected via a storage service network 4 configured, for example, froma Fibre Channel, Ethernet (registered trademark), InfiniBand or wirelessLAN (Local Area Network), and the respective storage nodes 3 areconnected via a backend network 5 configured, for example, from a LAN,Ethernet (registered trademark), InfiniBand or wireless LAN.

However, the storage service network 4 and the backend network 5 may beconfigured from same network, and the respective compute nodes 2 and therespective storage nodes 3 may also be connected to a management networkother than the storage service network 4 and the backend network 5.

The compute node 2 is a general purpose computer device which functionsas a host (higher-level device) to the storage node 3. Note that thecompute node 2 may also be a virtual computer device such as a virtualmachine. The compute node 2 reads and writes data from and into thestorage node via the storage service network based on the user'soperation or according to a request from an installed applicationprogram.

The storage node 3 is a service device which provides a storage area tothe compute node 2 for reading and writing data. The storage node 3 mayalso be a virtual machine. Moreover, the storage node 3 may also beconfigured to reside in the same physical node as the compute node 2. Inthe case of this embodiment, the respective storage nodes 3 arecollectively managed as a group referred to as a cluster 6 together withone or more other storage nodes 3 as shown in FIG. 1 . While the exampleof FIG. 1 illustrates a case where only one cluster 6 is set, aplurality of clusters 6 may also be set in the information processingsystem 1. The cluster 6 may also be referred to as a “distributedstorage system”.

The storage node 3 comprises, as shown in FIG. 2 , one or more CPUs(Central Processing Units) 10, one or more memories 11, a plurality ofstorage devices 12, and one or more first and second communicationdevices 13, 14. The storage node 3 is configured from a general purposephysical server device in which the CPU 10 and the storage device 12,and the first and second communication devices 13, 14, are connected viaan internal network 15.

The CPU 10 is a processor which governs the operational control of theoverall storage node 3. Moreover, the memory 11 is configured from avolatile semiconductor memory such as an SRAM (Static RAM (Random AccessMemory)) or a DRAM (Dynamic RAM), or a non-volatile semiconductormemory, and is used as a work memory of the CPU 10 for temporarilystoring various programs and necessary data. By executing the programsstored in the memory 11 with at least one or more CPUs 10, various typesof processing as the overall storage node 3 are executed as describedlater.

The storage device 12 is configured from a large capacity non-volatilestorage device such as an HDD (Hard Disk Drive), an SSD (Solid StateDrive) or an SCM (Storage Class Memory), is connected via an interfacesuch as NVMe (Non-Volatile Memory Express), SAS (Serial Attached SCSI(Small Computer System Interface)), or SATA (Serial ATA (AdvancedTechnology Attachment)), and provides a storage area for reading andwriting data according to a read request or a write request from thecompute node 2.

The first communication device 13 is an interface for the storage node 3to communicate with the compute node 2 via the storage service network4, and is configured, for example, from a Fibre Channel card, anEthernet (registered trademark) card, an InfiniBand card, or a wirelessLAN card. The first communication device 13 performs protocol controlduring communication with the compute node 2.

The second communication device 14 is an interface for the storage node3 to communicate with other storage nodes via the backend network 5, andis configured, for example, from a Fibre Channel card, an Ethernet(registered trademark) card, an InfiniBand card, a wireless LAN card, ora PCIe host adapter. The second communication device 14 performsprotocol control during communication with other storage nodes 3.

Note that, in the case of this embodiment, the respective storage nodes3 are collectively managed as a group referred to as a cluster 6together with one or more other storage nodes 3 as shown in FIG. 1 .While the example of FIG. 1 illustrates a case where only one cluster 6is set, a plurality of clusters 6 may also be set in the informationprocessing system 1. The cluster 6 may also be referred to as adistributed storage system.

(2) Flow of Various Types of Processing in Information Processing System

(2-1) Programs and Information Stored in Memory of Each Storage Node

The flow of various types of processing in the information processingsystem 1 is now explained. Here, the programs and information stored inthe memory 11 (FIG. 2 ) of each storage node 3 in relation to thevarious functions are foremost explained with reference to FIG. 3 toFIG. 6 .

As shown in FIG. 3 , the memory 11 of each storage node 3 stores aplurality of control software (this is hereinafter referred to as the“control software”) 20, a plurality of configuration information 21prepared in correspondence with each control software 20, a redundancymodule 22, a cluster control module 23, a storage node management table24 and a control software management table 25.

The control software 20 is software which functions as a storagecontroller of an SDS (Software Defined Storage). The control software 20has a function of receiving a read request or a write request from thecompute node 2, and reading or writing data from or into thecorresponding storage device 12 (FIG. 2 ). The control software 20 mayalso be referred to as a “storage control unit” or a “storage controlprogram”.

In the case of this embodiment, as shown in FIG. 7 , the respectivecontrol software 20 loaded in the storage node 3 are managed as onegroup for redundancy (this is hereinafter referred to as the “redundancygroup”) 26 together with one or more other control software 20 which areeach loaded in other mutually different storage nodes 3.

Note that FIG. 7 shows a case of configuring the redundancy group 26with two control software 20, and, while the ensuing explanation will beprovided on the premise that the redundancy group 26 is configured fromtwo control software 20, the redundancy group 26 may also be configuredfrom three or more control software 20.

In the redundancy group 26, at least one control software 20 is set to astatus of being able to receive a read request or a write request fromthe compute node 2 (status of the active system; this is hereinafterreferred to as the “active mode”), and the remaining control software 20are set to a status of not being able to receive a read request or awrite request from the compute node 2 (status of the standby system;this is hereinafter referred to as the “passive mode”).

Accordingly, a redundancy group 26 configured from two control software20 will take on either the configuration where the two control software20 are both set to the active mode (this is hereinafter referred to asthe “active-active configuration”), or the configuration where onecontrol software 20 is set to the active mode and the other controlsoftware 20 is set to the passive mode as a backup (this is hereinafterreferred to as the “active-passive configuration”).

With a redundancy group 26 that adopted the active-passiveconfiguration, when a failure occurs in the control software 20 set tothe active mode or the storage node 3 on which that control software 20is running, or that storage node 3 is removed from the cluster 6 (i.e.,the storage node 3 is decreased), the status of the control software 20that was previously set to the passive mode is switched to the activemode. Consequently, when the control software 20 that was set to theactive mode can no longer be operated, the I/O (Input/Output) processingthat was being executed by that control software 20 can be succeeded bythe control software 20 that was previously set to the passive mode(failover function).

In order to realize this kind of failover function, the plurality ofcontrol software 20 belonging to the same redundancy group 26 constantlyretain the configuration information 21 of the same contents. Theconfiguration information 21 is information which is required for thecontrol software 20 to execute various functions including the foregoingcapacity virtualization function, a tiering (hierarchical storagecontrol) function of migrating data having a high access frequency to astorage area having a faster response speed, a deduplication function ofdeleting duplicate data among the stored data, a compression function ofcompressing and storing data, a Snapshot function of retaining thestatus of data at a certain point in time, and a remote copy function ofsynchronously or asynchronously copying data to a remote location as adisaster countermeasure.

When the configuration information 21 of the active mode controlsoftware 20 configuring the redundancy group 26 is updated, a differencein the configuration information 21 before and after such update istransferred as difference data to the other control software 20configuring the redundancy group 26, and the other control softwareupdate their own configuration information 21 based on the differencedata. The configuration information 21 retained in the respectivecontrol software 20 configuring the redundancy group 26 is therebyconstantly maintained in a synchronous status.

As a result of the two control software 20 configuring the redundancygroup 26 constantly retaining the same configuration information 21,even when a failure occurs in the control software 20 set to the activemode or the storage node 3 loaded with the control software 20, or suchstorage node 3 is removed, the processing that was being executed bythat control software 20 can be immediately succeeded by another controlsoftware 20 within the same redundancy group 26.

The redundancy module 22 is a program with the function of constantlysynchronizing the configuration information 21 retained in therespective control software 20 in the same redundancy group 26. Ineffect, when any one of the control software 20 in its own storage node3 updates its own configuration information 21, the redundancy module 22sends, as difference data, a difference in the configuration information21 before and after such update to the storage node 3 loaded with theother control software 20 belonging to the same the redundancy group 26.Here, the redundancy module 22 designates the control software 20 inwhich the configuration information 21 should be updated based on thedifference data. Furthermore, the redundancy module 22 of the storagenode 3 that received the difference data updates the configurationinformation 21 retained in the designated control software 20 based onthe difference data. The redundancy module 22 may also be referred to as“redundancy software” or a “redundancy program”.

The cluster control module 23 is a program with the function oftransferring the I/O (Input/Output) request, which is configured from awrite request or a read request issued from the compute node 2, to thecluster control module 23 of the corresponding storage node 3 via thebackend network 5, or delivering the foregoing I/O request transferredfrom another cluster control module 23 to the control software 20 of thecorresponding redundancy group 26. The cluster control module 23 mayalso be referred to as “cluster control software” or a “cluster controlprogram”.

In effect, in the case of this embodiment, for each redundancy group 26,one or more virtual logical volumes (these are hereinafter referred toas the “virtual volumes”) are defined in correspondence with thatredundancy group 26, and these virtual volumes are provided to thecomputer node 2 as the storage area for reading/writing data.

In the foregoing case, the storage area in the virtual volume is managedby being partitioned into small areas of a predetermined size (these arehereinafter referred to as the “logical pages”). Moreover, the storagearea provided by the respective storage devices 12 equipped in thestorage node 3 is managed by being partitioned into small areas havingthe same size as the logical pages (these are hereinafter referred to asthe “physical pages”). However, the logical pages and the physical pagesdo not need to be the same size.

Furthermore, when the compute node 2 is to read or write data from orinto the intended virtual volume, the computer node 2 sends, to any oneof the storage nodes 3 in the corresponding cluster 6, an I/O requestwhich designates the identifier (LUN: Logical Unit Number) of thevirtual volume as the read/write destination of that data, the toplogical page of the read/write destination of that data in the virtualvolume, and the data length of that data.

Consequently, the cluster control module 23 of the storage node 3 thatreceived the foregoing I/O request transfers that I/O request, via thebackend network 5, to the cluster control module 23 of the respectivestorage nodes 3, in which the control software 20 of the redundancygroup 26 is associated with the read/write destination virtual volumedesignated in the received I/O request.

Moreover, the cluster control module 23 that received the foregoing I/Orequest delivers that I/O request to the control software 20 of theredundancy group 26 associated with the virtual volume that wasdesignated as the read/write destination in the I/O request.Subsequently, among the control software 20 to which the I/O request wasdelivered, the control software 20 set to the active mode, in responseto the I/O request, dynamically assigns one of the physical pages to thelogical page that was designated as the data write destination in theI/O request when the I/O request is a write request, and reads/writesdata from/into the physical page that was assigned to the logical page.

As means for executing the foregoing series of processing, the clustercontrol module 23 is sharing information such as in which storage node 3each control software 20 existing in the cluster 6 has been deployed, towhich redundancy group 26 each control software 20 belongs, and whethereach control software 20 has been set to the status of an active mode ora passive mode, and stores and manages the foregoing information in thecontrol software management table 25 described later.

Meanwhile, the configuration information 21 is configured frominformation required for the control software 20 to realize variousfunctions such as the capacity virtualization function, the tiering(hierarchical storage control) function, the Snapshot function, theremote copy function, the deduplication function, and the compressionfunction described above.

FIG. 4 shows a logical-physical conversion table 27 which configures apart of the configuration information 21. The logical-physicalconversion table 27 is a table that is used for managing the correlationof the foregoing logical pages and the physical pages in relation to thecapacity virtualization function, and is configured by comprising, asshown in FIG. 4 , a LUN column 27A, a logical page column 27B and aphysical page column 27C. In the logical-physical conversion table 27,one line corresponds to one logical page.

The LUN column 27A stores the LUN of the virtual volume that wasassigned by the cluster 6 to each virtual volume to be provided to thecompute node 2, and the logical page column 27B stores the identifier(logical page number) that is assigned to each logical page in thecorresponding virtual volume.

Moreover, the physical page column 27C is separated into a node IDcolumn 27CA, a drive number column 27CB and a sector number column 27CC.The node ID column 27CA stores an identifier (node ID) that is unique tothe storage node 3 which was assigned to that storage node whichprovides the physical page assigned to the corresponding logical page,and the drive number column 27CB stores an identifier (drive number)that is unique to the storage device 12 which was assigned to thatstorage device 12 (FIG. 2 ) which provides the physical page PP.Furthermore, the sector number column 27CC stores an identifier (sectornumber) that is unique to the sector which was assigned to that storagearea (sector) corresponding to the physical page PP in the correspondingstorage device 12.

Accordingly, the case of FIG. 4 shows that, for instance, a storage area(sector) having a sector number of “0x00010000” in the storage device 12having a drive number of “1” in the storage node 4 having a node ID of“0” has been assigned as the physical page to the logical page having alogical page number of “0” of the virtual volume having a LUN of “1”.

Moreover, the storage node management table 24 is a table that is usedfor managing the load status of the respective storage nodes 3 in thesame cluster 6, and is configured by comprising, as shown in FIG. 5 , anode ID column 24A, a status column 24B, a capacity column 24C, a usedcapacity column 24D, a CPU load column 24E, a memory amount column 24F,a used memory amount column 24G, a communication band column 24H, a usedcommunication band column 24I and a failure group ID column 24J. In thestorage node management table 24, one line corresponds to one storagenode 3.

The node ID column 24A stores the node ID of the respective storagenodes 3 configuring the cluster 6, and the status column 24B stores thecurrent status (“normal” or “abnormal”) of the corresponding storagenode 3. Moreover, the capacity column 24C stores the capacity of allstorage devices of the corresponding storage node 3, and the usedcapacity column 24D stores the current used capacity of the storagedevice in the corresponding storage node 3.

Furthermore, the CPU load column 24E stores the current utilization ofthe CPU 10 (FIG. 2 ) in the corresponding storage node 3, the memoryamount column 24F stores the capacity of the memory 11 (FIG. 2 ) in thatstorage node 3, and the used memory amount column 24G stores the currentused amount of the memory 11 in the corresponding storage node 3.

Furthermore, the communication band column 24H stores the size of thecommunication band of the backend network 5 that is available to thecorresponding storage node 3, and the used communication band column 24Istores the band amount of the backend network 5 that is currently beingused by that storage node 3 for communicating with other storage nodes3.

Furthermore, the failure group ID column 24J stores the group ID of thefailure group belonging to the corresponding storage node 3. Here, theterm “failure group” refers to the group of the storage nodes 3 thatwill be subject to a similar failure when a failure occurs in the powersource or the like, such as when the same power source is being shared.

The information stored in the storage node management table 24 isinformation that is shared by the cluster control modules 23 of allstorage nodes 3 configuring the same cluster 6. Accordingly, as meansfor sharing the same information among the cluster control modules 23 ofthe respective storage nodes 3 in the same cluster 6, one clustercontrol module 23 is selected based on a predetermined method, as therepresentative cluster control module 23, among the cluster controlmodules 23 equipped in the respective storage nodes 3 configuring thecluster 6.

Subsequently, the representative cluster control module 23 periodicallycollects necessary information from the cluster control modules 23 ofother storage nodes 3, and updates the storage node management table 24,which it is managing, based on the collected information. Moreover, therepresentative cluster control module 23 transfers the collectedinformation to the cluster control modules 23 of the respective storagenodes 3 in that cluster 6 and thereby causes such cluster controlmodules 23 to update their own storage node management table 24 to thelatest status.

However, the configuration may also be such that the cluster controlmodels 23 of the respective storage nodes 3 periodically send necessaryinformation to the cluster control models 23 of all other storage nodes3 in the same cluster 6, and the cluster control modules 23 of therespective storage nodes 3 update the storage node management table 24in their own node based on the received information.

The control software management table 25 is a table that is used for thecluster control module 23 to manage all control software 20 existing inthe cluster 6, and is configured by comprising, as shown in FIG. 6 , acontrol software ID column 25A, a status column 25B, a group ID column25C, an operating node ID column 25D, a used capacity column 25E, a CPUload column 25F, a used memory amount column 25G, a used communicationband column 25H and a LUN column 25I. In the control software managementtable 25, one line corresponds to one control software 20.

The control software ID column 25A stores an identifier (controlsoftware ID) that is unique to the control software 20 which wasassigned to that control software 20 which is currently running on thecorresponding cluster 6, and the status column 25B stores the currentstatus of the corresponding control software 20. As the status of thecontrol software 20, there are, in addition to the foregoing active mode(“Active”) and passive mode (“Passive”), dead (“Dead”) as an inoperablestatus caused by a failure or the like, and stop (“Stop”) as a statuswhere the I/O processing is stopped for maintenance or the like.

Moreover, the group ID column 25C stores an identifier (group ID) thatis unique to the redundancy group 26 which was assigned to thatredundancy group 26 (FIG. 7 ) configuring the corresponding controlsoftware 20, and the operating node ID column 25D stores the node ID ofthe storage node 3 in which the corresponding control software 20 isrunning. Moreover, the used capacity column 25E stores the totalcapacity of the storage areas that are currently being used by thecorresponding control software 20.

Moreover, the CPU load column 25F stores the current utilization of theCPU 10 (FIG. 2 ) for running the corresponding control software 20, andthe used memory amount column 25G stores the capacity of the memory 11(FIG. 2 ) which is currently being used for the corresponding controlsoftware 20. Furthermore, the used communication band column 25H storesthe band amount of the backend network 5 that is currently being used bythe corresponding control software 20 for communicating with otherstorage nodes 3.

The LUN column 25I stores the LUN (Logical Unit Number: volume number)of the identification information for identifying the virtual volumeprovided to the compute node 2, and is managed by being associated withthe control software ID 25A of the control software which manages thevirtual volume identified by the LUN.

When the cluster control module 23 of each storage node 3 receives anI/O request (read request or write request) from the compute node 2, thecluster control module 23 acquires the LUN included in the I/O request,and uses the control software management table 25 to identify thecontrol software ID 25A and the operating node ID associated with thatLUN. Consequently, the cluster control module 23 of each storage node 3is able to identify the control software which is managing the virtualvolume to become the I/O request destination, and the deploymentdestination node of that control software. Note that the LUN of the LUNcolumn 25I may also be managed by being associated with the group ID 25Cof the redundancy group corresponding to the virtual volume that isidentified with the LUN. If the group ID can be identified from the LUNincluded in the I/O request based on the control software managementtable 25, it will also be possible to identify the control software IDincluded in the redundancy group identified with the group ID, and thecontrol software which is identified with the control software ID.

The information stored in the control software management table 25 isalso information which is shared by the cluster control modules 23 ofall storage nodes 3 configuring the same cluster 6. Thus, therepresentative cluster control module 23 periodically collects necessaryinformation from the cluster control modules 23 of other storage nodes3, and updates the control software management table 25, which it ismanaging, based on the collected information. Moreover, therepresentative cluster control module 23 transfers the collectedinformation to the cluster control modules 23 of the respective storagenodes 3 in that cluster 6 and thereby causes such cluster controlmodules 23 to update their own storage node management table 24 to thelatest status.

(2-2) Flow of Various Types of Processing

(2-2-1) Flow of Write Processing

FIG. 7 shows the flow of processing in the information processing system1 when a write request is given to one of the storage nodes 3 in thecluster 6 from the compute node 2 during a normal period.

In FIG. 7 , “storage node A” to “storage node D” respectively representthe storage node 3, and “control software A” to “control software H”respectively represent the control software 20. Furthermore, FIG. 7shows a case where the “control software A” running on the “storage nodeA” and the “control software B” running on the “storage node B”configure a redundancy group 26, the “control software C” running on the“storage node B” and the “control software D” running on the “storagenode C” configure a redundancy group 26, the “control software E”running on the “storage node C” and the “control software F” running onthe “storage node D” configure a redundancy group 26, and the “controlsoftware G” running on the “storage node D” and the “control software H”running on the “storage node A” configure a redundancy group 26.Moreover, here, let it be assumed that the configuration of therespective redundancy groups 26 created based on the redundancy of thecontrol software 20 is an active-passive configuration.

In the foregoing case, during a normal period, the cluster controlmodules 23 of the respective storage nodes 3 share information such asthe load status of the CPU 10 (FIG. 2 ) and the used amount of thememory 11 (FIG. 2 ) of the respective storage nodes 3 described above,and store and manage such information in the storage node managementtable 24 and the control software management table 25 (S1).

Subsequently, when a write request from the compute node 2 is given toany one of the storage nodes 3 in the cluster 6, the cluster controlmodule 23 of the storage node 3 uses the control software managementtable 25 and transfers the write request to each control software 20 ofthe corresponding redundancy group 26 via the cluster control module 23of the corresponding storage node 3 based on the LUN included in thewrite request as described above. Among the control software 20 to whichthe write request was transferred, the control software 20 set to theactive mode executes the write processing. Note that the cluster controlmodules 23 may transfer the write request only to the control software20 set to the active mode.

Here, let it be assumed that the write request indicates, as the writedestination, the virtual volume associated with the redundancy group 26configured from the “control software A” and the “control software B”.Moreover, in the redundancy group 26, let it be assumed that the“control software A” is set to the active mode, and the “controlsoftware B” is set to the passive mode. Accordingly, in the foregoingcase, the “control software A” set to the active mode will execute thewrite processing according to the write request.

When the foregoing write processing is executed, the redundancy module22 of the “storage node A” loaded with the “control software A”, whichexecuted the write processing, acquires, from the cluster controlmodules 23 in the same storage node 3 (“storage node A”), the controlsoftware ID of the control software 20 (“control software B”)configuring the redundancy group 26 together with the “control softwareA” that executed the write processing, and the IP address on the backendnetwork 5 of the storage node 3 (“storage node B”) loaded with thecontrol software 20 (S2).

Moreover, the redundancy module 22 of the “storage node A” updates theconfiguration information 21 retained in the “control software A”according to the processing contents of the write processing (S3), andsends, via the backend network 5, the difference data representing thedifference before and after the update of the configuration information21 to the storage node 3 (“storage node B”) of the IP address acquiredin step S2 (S4). Here, the redundancy module 22 designates the “controlsoftware B”, which configures the redundancy group 26 together with the“control software A”, as the control software 20 to update theconfiguration information 21 based on the difference data.

Consequently, the redundancy module 22 of the “storage node B” thatreceived the difference data updates the configuration information 21retained in the “control software B” based on the difference data in thesame manner as the configuration information 21 retained in the “controlsoftware A”.

Accordingly, the two configuration information 21 retained respectivelyin the two control software 20 (“control software A” and “controlsoftware B”) configuring the same redundancy group 26 are maintained tohave the same contents.

(2-2-2) Flow of Processing Upon Occurrence of Failure in Storage Nodes

The flow of processing when a failure occurs in any one of the storagenodes 3 configuring the cluster 6 is now explained. Note that, here, asshown in FIG. 8 , explained is the flow of processing in a case where afailure occurred in the “storage node A”, and the “control software A”and the “control software H” loaded in the “storage node A” becomeinoperable.

In the foregoing case, any one of the cluster control modules 23 thatdetected the foregoing failure notifies such failure to therepresentative cluster control module 23. Here, let it be assumed thatthe cluster control module 23 of the “storage node B” has been selectedas the representative cluster control module 23, and that the notice hasbeen sent to the cluster control module 23 of the “storage node B”.

The representative cluster control module 23 that received the noticerefers to the control software management table 25 (FIG. 6 ), and sendsa notice to the effect that a failure has occurred in the “controlsoftware A” to the cluster control module 23 of the “storage node B”loaded with the “control software B” configuring the same redundancygroup 26 as the “control software A”. Moreover, the representativecluster control module 23 sends a notice to the effect that a failurehas occurred in the “control software H” to the to the cluster controlmodule 23 of the “storage node D” loaded with the “control software G”configuring the same redundancy group 26 as the “control software H”(S10).

The cluster control module 23 of the “storage node B” that received thenotice changes the status of the “control software B”, which waspreviously set to the passive mode, to the active mode (S11).Consequently, the I/O processing that was previously being executed bythe “control software A” will be succeeded by the “control software B”.

Moreover, the cluster control module 23 of the “storage node D” thatreceived the notice sends a command to the redundancy module 22, whichis associated with the “control software G”, to the effect of refrainingfrom transferring the difference data to the “storage node A” even whenthe configuration information 21 retained in the “control software G” issubsequently updated (S12).

Subsequently, in order to recover the redundancy configuration of the“control software B” and the “control software G”, the representativecluster control module 23 determines the storage node 3 to become thenew deployment destination of the “control software A” and the storagenode 3 to become the new deployment destination of the “control softwareH”, respectively, as shown in FIG. 9 , and notifies the determinationresult to the cluster control modules 23 in the respective storage nodes3 (S13).

Here, the representative cluster control module 23 selects as the newdeployment destinations of the “control software A” and the “controlsoftware H”, by referring to the storage node management table 24 (FIG.5 ), a storage node 3 which is different from the storage nodes 3 inwhich the other control software 20 configuring the same redundancygroup 26 has previously been deployed, and which belongs to a failuregroup that is different from the storage node 3 in which the othercontrol software 20 configuring the same redundancy group 26 haspreviously been deployed. Moreover, when there are a plurality of suchstorage nodes 3 (storage nodes 3 that could be candidates of the newdeployment destinations of the “control software A” and the “controlsoftware H”), the representative cluster control module 23 refers to thestorage management table 24 and the control software management table25, and determines the storage nodes 3 having a low load as the newdeployment destinations of the “control software A” and the “controlsoftware H”, respectively.

Incidentally, the term “storage nodes 3 having a low load” refers to astorage node 3 having the lowest CPU load among the candidate storagenodes 3 in which the capacity of the storage device, the CPU load, thememory amount, and the communication band of the backend network 5 ofthat storage node 3 do not exceed the tolerance level even when thecontrol software 20 is deployed. Here, let it be assumed that the newdeployment destination of the “control software A” has been determinedto be the “storage node C”, and the new deployment destination of the“control software H” has been determined to be the “storage node B”,respectively. Incidentally, the term “tolerance level” refers to a levelthat is within the range of the capacity and memory amount of thestorage device of the storage node 3 with regard to the capacity and thememory amount, 100% or less with regard to the CPU load, and within therange of the available communication band with regard to the backendnetwork 5. Otherwise, when thresholds have been set in advance for suchcapacity, CPU load, memory amount and/or communication band of thebackend network 5, the term “tolerance level” refers to the range withinsuch thresholds. The same shall apply in the ensuing explanation.

The cluster control module 23 of the “storage node B” that received thenotice issues a command to the redundancy module 22 in its own storagenode 3 for fully copying the configuration information 21 retained inthe “control software B” to the “storage node C” as the new deploymentdestination of the “control software A” (S14). Moreover, the redundancymodule 22 that received the foregoing command transfers theconfiguration information 21 to the “storage node C” via the backendnetwork 5 (S15). Subsequently, the cluster control module 23 of the“storage node C” that received the configuration information 21 storesthe received configuration information 21 in the memory 11 (FIG. 2 ) byassociating it with the “control software A”.

When the full copy of the configuration information 21 retained in the“control software B” to the “storage node C” is complete, the redundancymodule 22 of the “storage node B” subsequently starts to transfer thedifference data, which shows the difference in comparison to theoriginal configuration information 21, to the “storage node C” each timethat the configuration information 21 retained in the “control softwareB” is updated, and, consequently, the configuration information 21retained in the “control software B” and the configuration information21 copied to the “storage node C” are synchronously updated (S16).

Similarly, the cluster control module 23 of the “storage node D” thatreceived the foregoing notice issues a command to the redundancy module22 in its own storage node 3 of fully copying the configurationinformation 21 retained in the “control software G” to the “storage nodeB” as the new deployment destination of the “control software H” (S17).Moreover, the redundancy module 22 that received the foregoing commandtransfers the configuration information 21 to the “storage node B” viathe backend network 5 (S18). Subsequently, the cluster control module 23of the “storage node B” that received the configuration information 21stores the received configuration information 21 in the memory 11 byassociating it with the “control software H”.

Even when the full copy of the configuration information 21 retained inthe “control software G” to the “storage node B” is complete, theredundancy module 22 of the “storage node D” subsequently starts totransfer the difference data, which shows the difference in comparisonto the original configuration information 21, to the “storage node B”each time that the configuration information 21 retained in the “controlsoftware G” is updated, and, consequently, the configuration information21 retained in the “control software G” and the configurationinformation 21 copied to the “storage node B” are synchronously updated(S19).

Subsequently, the cluster control module 23 of the “storage node C”activates the control software 20 installed in the “storage node C” inthe passive mode as the new “control software A” (S20). Similarly, thecluster control module 23 of the “storage node B” activates the controlsoftware 20 installed in the “storage node B” in the passive mode as thenew “control software H” (S21). The redundancy configuration of the“control software B” and the “control software G” is thereby recovered.

The representative cluster control module 23 subsequently updates thecontrol software management table 25, which it is managing, according tothe foregoing configuration change. Specifically, the representativecluster control module 23 changes the node ID stored in the operatingnode ID column 25D (FIG. 6 ) of the line corresponding to the “controlsoftware A” in the control software management table 25 to the node IDof the “storage node C” as the new deployment destination of the“control software A”, and changes the node ID stored in the operatingnode ID column 25D of the line corresponding to the “control software H”in the control software management table 25 to the node ID of the“storage node B” as the new deployment destination of the “controlsoftware H”.

Moreover, the representative cluster control module 23 notifies thecontents of the thus updated control software management table 25 thatit is managing to the cluster control modules 23 of the respectivestorage nodes 3 in the same cluster (S22). Consequently, each clustercontrol module 23 that received the foregoing notice updates the controlsoftware management table 25, that it is managing, to be the samecontents as the contents of the new control software management table 25notified from the representative cluster control module 23.

The series of processing performed when a failure occurs in the “storagenode A” is thereby ended.

Note that, in the foregoing example, as shown in FIG. 9 , theconfiguration will be such that the three control software 20 of“control software B”, “control software C” and “control software H” runon the “storage node B”, and the three control software 20 of “controlsoftware A”, “control software D” and “control software E” run on the“storage node C”. However, in order to suppress the CPU load and thememory consumption, by merging the configuration information 21 of thecontrol software 20 and causing certain control software 20 to succeedthe I/O processing the other control software 20, the number of controlsoftware 20 to run on the storage node 3 may be reduced.

For instance, in the example of FIG. 9 , the configuration information21 retained in the “control software C” can be merged with theconfiguration information 21 retained in the “control software B”, theI/O processing of the “control software C” can be succeeded by the“control software B”, and the “control software C” and the “controlsoftware D” belonging to the same redundancy group can be stopped.

Moreover, the flow of processing that is executed in the cluster 6 upondecreasing the storage node 3 of the cluster 6 (removing any one of thestorage nodes 3) is also basically the same as above. However, in theforegoing case, the administrator of the information processing system 1designates in advance the storage node 3 to be removed, and issues, tothe representative cluster control module 23, a command to the effectthat all control software 20 running on the storage node 3 should bemigrated to other storage nodes 3 in the same cluster 6 (this ishereinafter referred to as the “all control software migrationcommand”).

The representative cluster control module 23 that received the allcontrol software migration command executes the processing of step S11onward according to the all control software migration command uponexchanging the “storage node 3 to be removed” and the foregoing “storagenode 3 that was subject to a failure”. Consequently, the administratorcan remove, from the cluster 6, the storage node 3 to be removed afterthe processing up to foregoing step S22 is completed.

(2-2-3) Flow Upon Adding Storage Node

The flow of processing that is executed in the cluster 6 when a storagenode 3 referred to as the “storage node E” is to be newly added to thecluster 6, as shown in FIG. 10 , from the status of FIG. 7 , is nowexplained.

Foremost, when the “storage node E” is added and the cluster controlmodule 23 loaded in the “storage node E” is thereafter activated,information such as the CPU load and the memory consumption of the“storage node E” is notified from the cluster control module 23 to thecluster control modules 23 of the respective storage nodes 3 in thecluster 6.

Subsequently, the foregoing representative cluster control module 23that received the foregoing notice (cluster control module 23 of the“storage node B”) determines the control software 20 to be redeployed inthe “storage node E”, which was added among the “control software A” tothe “control software H”, which were loaded in any one of the storagenodes 3 among the “storage node A” to the “storage node D”, for the loadleveling (rebalance) of the respective storage nodes 3. Therepresentative cluster control module 23 notifies the determinationresult to the respective cluster control modules 23 in the cluster 6(S30).

Here, based on the CPU load and the unused memory amount of the “storagenode A” to the “storage node E” and the size of the unused band of thebackend network 5, the representative cluster control module 23determines the control software 20 to be redeployed in the “storage nodeE” so as to level the foregoing CPU load and so on, and so that aplurality of control software 20 of the active mode will not run on thesame storage node 3 to the extent possible. Here, let it be assumed thatthe “control software H” loaded in the “storage node A” has beendetermined as the control software 20 to be redeployed in the “storagenode E”.

Meanwhile, when the cluster control module 23 of the “storage node D”loaded with the “storage node D”, which configures the redundancy group26 together with the “control software H”, receives the foregoingdetermination result, the cluster control module 23 issues a command tothe redundancy module 22 of fully copying the configuration information21 of the active mode “control software G” in the redundancy group 26 tothe “storage node E” (S31). Moreover, the redundancy module 22 thatreceived the foregoing command transfers the configuration information21 of the “control software G” to the “storage node E” via the backendnetwork 5 (S32). Consequently, the cluster control module 23 of the“storage node E” that received the foregoing configuration information21 stores the received configuration information 21 in the memory 11.

Moreover, when the redundancy module 22 of the “storage node D”completes copying the configuration information 21 of the “controlsoftware G” to the “storage node E”, the redundancy module 22 starts totransfer the difference data, which shows the difference in comparisonto the original configuration information 21 before and after theupdate, to the “storage node E” each time that the configurationinformation 21 retained in the “control software G” is updated.Consequently, the copy result of the configuration information 21 (thatis, the configuration information 21) of the “control software G” in the“storage node E” is updated based on the difference data, and thesynchronization of the configuration information 21 of the “controlsoftware G” and the copy result of the configuration information 21 ofthe “control software G” in the “storage node E” is thereby started.

Subsequently, the cluster control module 23 of the “storage node D”issues a command to the redundancy module 22 so that the redundancydestination of the “control software G” will only be the “storage nodeE” as shown in FIG. 11 (S33). Moreover, the cluster control module 23 ofthe “storage node D” notifies the cluster control module 23 of the“storage node E” to the effect that the full copy of the configurationinformation 21 retained in the “control software G” to the “storage nodeE” is complete.

Consequently, the cluster control module 23 of the “storage node E” thatreceived the foregoing notice activates the control software 20installed in the “storage node E” in the passive mode as the new“control software H” (S34). Moreover, the cluster control module 23 ofthe “storage node E” notifies the cluster control module 23 of the“storage node A”, on which the original “control software H” is running,to such effect.

Subsequently, the cluster control module 23 of the “storage node A” thatreceived the foregoing notice stops the operation of the “controlsoftware H” that is running on its own storage node 3 (S35). Theredeployment of the “control software H” in the “storage node E” isthereby completed.

Subsequently, the representative cluster control module 23 updates thecontrol software management table 25, which it is managing, according tothe configuration change of the system as described above. Specifically,the representative cluster control module 23 changes the node ID storedin the operating node ID column 25D (FIG. 6 ) of the line correspondingto the “control software H” in the control software management table 25to the node ID of the “storage node E” as the new deployment destinationof the “control software H”.

Moreover, the representative cluster control module 23 notifies thecontents of the thus updated control software management table 25, whichis it managing, to the cluster control models 23 of the respectivestorage nodes 3 in the same cluster 6 (S36). Consequently, therespective cluster control modules 23 that received the foregoing noticeupdate the control software management table 25, that they are managing,to be the same contents as the contents of the new control softwaremanagement table 25 notified from the representative cluster controlmodule 23.

Meanwhile, the representative cluster control module 23 subsequentlyrefers to the storage node management table 24 and the control softwaremanagement table 25, and, when the CPU load and unused memory amount ofthe “storage node E” and the communication band of the backend network 5are sufficient, as shown in FIG. 12 , issues a command to the clustercontrol module 23 of the “storage node E” so as to activate the newcontrol software 20 as the “control software I” in the active mode.Consequently, the cluster control module 23 of the “storage node E” thatreceived the foregoing activation command of the “control software I”activates unused control software 20 in the “storage node E” in theactive mode as the “control software I” (S37).

Moreover, the representative cluster control module 23 determines thestorage node 3 of the deployment destination of the control software 20(this is hereinafter referred to as the “control software J”) whichconfigures the same redundancy group 26 as the “control software I”.Here, the representative cluster control module 23 refers to the storagemanagement table 24 and the control software management table 25, anddetermines the storage nodes 3 having a low load as the deploymentdestination of the “control software J”. Here, let it be assumed thatthe “storage node A” was decided as the deployment destination of the“control software J”.

Subsequently, the representative cluster control module 23 issues acommand to the cluster control module 23 of the thus determined storagenode 3 (“storage node A”) to activate the new control software 20 as the“control software J” in the passive mode. Consequently, the clustercontrol module 23 of the “storage node A” that received the foregoingcommand activates the new control software 20 in the “storage node A” inthe passive mode as the “control software J” (S38).

Subsequently, the representative cluster control module 23 notifies thenode ID of the storage node 3 loaded with the “control software J” tothe cluster control module 23 of the “storage node E”. Moreover, thecluster control module 23 of the “storage node E” that received theforegoing notice notifies the redundancy module 22 in its own storagenode 3 of the control software ID of the control software 20 (“controlsoftware J”), which configures the redundancy group 26 as the “controlsoftware I”, and the node ID of the storage node 3 (“storage node A”)loaded with the control software 20 (S39). Note that, in the ensuingexplanation, the notice given to the redundancy module 22 in its ownstorage node 3 from the cluster control module 23 of the “storage nodeE” is referred to as an “ID notice”.

Moreover, the representative cluster control module 23 updates thecontrol software management table 25, which it is managing, according tothe configuration of the new redundancy group 26 as described above.Specifically, the representative cluster control module 23 newlyregisters the “control software l” and the “control software J” in thecontrol software management table 25.

Moreover, the representative cluster control module 23 stores “Active”in the control software management table 25 of the line corresponding tothe “control software l” in the control software management table 25,stores the group ID assigned to the redundancy group 26, which isconfigured by the “control software I”, in the group ID column 25C (FIG.6 ) of that line, and additionally stores the node ID of the “storagenode E”, which is the deployment destination of the “control softwareI”, in the operating node ID column 25D of that line.

Furthermore, the representative cluster control module 23 stores“Passive” in the status column 25B (FIG. 6 ) of the line correspondingto the “control software J” in the control software management table 25,stores the group ID assigned to the redundancy group 26, which isconfigured by the “control software J”, in the group ID column 25C (FIG.6 ) of that line, and additionally stores the node ID of the “storagenode A”, which is the deployment destination of the “control softwareJ”, in the operating node ID column 25D of that line.

Meanwhile, the redundancy module 22 that received the foregoing IDnotice subsequently causes the “control software l” to start receivingread requests or write requests, and, by sending, as difference data,the difference in comparison to the original configuration information21 to the “storage node A” loaded with the “control software J” eachtime the configuration information 21 of the “control software l” ischanged, the configuration information 21 of the “control software J” ischanged by being constantly synchronized with the configurationinformation 21 of the “control software I” (S40).

Moreover, the representative cluster control module 23 notifies thecontents of the thus updated control software management table 25, thatit is managing, to the cluster control modules 23 of the respectivestorage nodes 3 in the same cluster 6 (S41). Consequently, therespective cluster control modules 23 that received the foregoing noticeupdate the control software management table 25, that they are managing,to be the same contents as the contents of the new control softwaremanagement table 25 notified from the representative cluster controlmodule 23.

Note that, in addition to the operation of immediately operating thecontrol software 20 on the added storage node 3 after the storage node 3is added as described above, it is also possible to apply an operationof leaving the storage node 3 unused without operating the controlsoftware 20 on the storage node 3 immediately after adding the storagenode 3 in preparation for the sudden increase in the load of the otherstorage nodes 3 in the cluster 6.

(2-2-4) Active-Active Configuration

FIG. 13 shows a logical configuration example of the cluster 6 in eachredundancy group 26 when the two control software 20 configuring thatredundancy group 26 are both set to the active mode.

In the foregoing case, in the respective redundancy groups 26, even whenany one of the control software 20 updates the configuration information21, a difference in the configuration information 21 before and aftersuch update is transferred as difference data to the storage node 3 onwhich other control software 20 is running, and the configurationinformation 21 retained in the other control software update is updatedbased on the difference data (S42). Consequently, the two configurationinformation 21 respectively retained in the two control software 20configuring the same redundancy group 26 are thereby constantlymaintained (synchronously updated) to have the same contents.

Because the flow of processing when a storage node 3 is subject to afailure or when a storage node 3 is added in cases where the redundancygroup 26 is set to have an active-active configuration is the same asthe case of the active-passive configuration described above withreference to FIG. 7 to FIG. 12 , the explanation thereof is omitted.

(2-3) Specific Processing Contents of Respective Programs DuringAddition/Removal of Storage Nodes

The specific processing contents of the representative cluster controlmodule 23 and the redundancy module 22 upon adding or removing thestorage node 3 are now explained. Note that, in the ensuing explanation,while a “program” such as the cluster control module 23 or theredundancy module 22 is explained as the processing subject of thevarious types of processing, it goes without saying that, in effect, theCPU 10 (FIG. 2 ) of the storage node 3 executes the processing based onthe “program”.

(2-3-1) Redundancy Recovery Processing

FIG. 14 shows the processing routine of the redundancy recoveryprocessing to be executed by the representative cluster control module23 in the cluster 6 when a failure occurs in any one of the storagenodes 3 in that cluster 6.

During a normal period, the representative cluster control module 23periodically communicates with the cluster control modules 23 loaded inthe respective nodes 3 in the cluster 6 to which the representativecluster control module 23 belongs, and collects, from the respectivecluster control modules 23, information to be stored in the storage nodemanagement table 24 described above with reference to FIG. 5 , and thecontrol software management table 25 described above with reference toFIG. 6 .

Moreover, the representative cluster control module 23 executes theredundancy recovery processing shown in FIG. 14 in parallel with theforegoing processing and, based on whether or not it was possible toperiodically communicate with the cluster control modules 23 of therespective storage nodes 3, monitors whether or not a failure hasoccurred in any one of the storage nodes 3, or monitors whether or not apredetermined command to the effect of removing any one of the storagenode 3 has been issued (S50).

When the representative cluster control module 23 obtains a positiveresult in step S50 as a result of detecting that failure occurred in oneof the storage nodes 3, for each control software (this is hereinafterreferred to as the “inoperable control software”) 20 deployed in thestorage node 3 that was subject to a failure, the representative clustercontrol module 23 issues a command, so as to cause the I/O processing ofthe inoperable control software 20 to be succeeded by another controlsoftware (this is hereinafter referred to as the “intragroup controlsoftware”) 20 which is configuring the same redundancy group 26 as theinoperable control software 20, to the cluster control module 23 of thestorage node 3 loaded with the intragroup control software 20 (S51).

Here, when there are a plurality of intragroup control software 20, therepresentative cluster control module 23 determines, as the controlsoftware 20 to succeed the I/O processing of the inoperable controlsoftware 20, the intragroup control software 20 among the foregoingplurality of intragroup control software 20 deployed in a storage node 3in which the capacity, CPU load, memory amount, and communication bandof the backend network 5 of the storage device will not exceed thetolerance level even when the inoperable control software 20 succeedsthe capacity, CPU load, memory amount, and communication band of thebackend network 5 of the storage device that was being used before thefailure occurred, and issues a command to the cluster control module 23of the storage node 3 loaded with the intragroup control software 20.

Consequently, the cluster control module 23 that received the foregoingcommand refers to the control software management table 25, and, whenthe intragroup control software 20 has been set to the passive mode,changes the status of that intragroup control software 20 to an activemode.

Subsequently, the representative cluster control module 23 executes thedeployment destination storage node determination processing fordetermining the storage node (this is hereinafter referred to as the“deployment destination storage node”) 3 to become the deploymentdestination of the new control software 20 in substitute for theinoperable control software 20 (S52). For example, when the redundancygroup 26 configured by the inoperable control software 20 and theintragroup control software 20 is an active-active configuration, therepresentative cluster control module 23 refers to the storage nodemanagement table 24 (FIG. 5 ), and determines a storage node 3 having alow load as the deployment destination storage node 3.

Next, the representative cluster control module 23 determines whether ornot it was possible to determine the deployment destination storage node3 based on the deployment destination storage node determinationprocessing of step S52 (S53). When the representative cluster controlmodule 23 obtains a negative result in the foregoing determination, therepresentative cluster control module 23 ends the redundancy recoveryprocessing.

Meanwhile, when the representative cluster control module 23 obtains apositive result in the determination of step S53, the representativecluster control module 23 notifies the node ID of the thus determineddeployment destination storage node 3 to the cluster control module 23of the storage node 3 loaded with the foregoing intragroup controlsoftware 20 (S54).

Consequently, the cluster control module 23 that received the foregoingnotice issues a command to the redundancy module 22 of its own storagenode 3 so as to transfer the configuration information 21 retained inthe intragroup control software 20 to the deployment destination storagenode 3. Moreover, the redundancy module 22 that received the foregoingcommand fully copies the configuration information 21 to the deploymentdestination storage node 3 by transferring the configuration information21 retained in the intragroup control software 20 to the deploymentdestination storage node 3 via the backend network 5.

When the configuration information 21 retained in the intragroup controlsoftware 20 is subsequently updated, the redundancy module 22 transfersthe difference data, which is a difference in the configurationinformation 21 before and after such update, to the deploymentdestination storage node 3 via the backend network 5. Consequently, onthe side of the deployment destination storage node 3, the copy of theconfiguration information 21 is updated based on the difference data,and the configuration information 21 retained in the intragroup controlsoftware 20 and the copy of the configuration information 21 in thedeployment destination storage node 3 will be completely synchronouslyupdated.

Meanwhile, after the processing of step S54, the representative clustercontrol module 12 waits for the synchronization of the configurationinformation 21 retained in the intragroup control software 20 and thecopy of the configuration information 21 in the deployment destinationstorage node 3 to be completed (S55). When the synchronization of thetwo configuration information 21 is eventually completed, therepresentative cluster control module 23 issues a command to the clustercontrol module 23 of the deployment destination storage node 3 so as toactivate the control software 20 to become the substitute of theinoperable control software 20 (S56).

Consequently, the cluster control module 23 of the deploymentdestination storage node 3 that received the foregoing command activatesunused control software 20. Here, the cluster control module 23activates the control software 20 in the passive mode upon adopting anactive-passive configuration for the redundancy group 26 configured bythe control software 20 and the intragroup control software 20, andactivates the control software 20 in the active mode upon adopting anactive-active configuration for that redundancy group 26. Moreover, thecluster control module 23 subsequently associates the control software20 with the configuration information which was fully copied to thedeployment destination storage node 3 in step S54. Consequently, thecontrol software 20 subsequently retains the configuration information21, and executes I/O processing based on the configuration information21.

Subsequently, the representative cluster control module 23 updates thecontrol software management table 25 (FIG. 6 ) retained in the ownstorage node 3 to the status after the inoperable control software 20has been replaced with the control software 20 activated in step S56(S57).

Specifically, the representative cluster control module 23 stores“Dead”, which means that the inoperable control software 20 can nolonger be used, in the status column 25B (FIG. 6 ) of the linecorresponding to the inoperable control software 20 in the controlsoftware management table 25 in its own storage node 3.

Moreover, the representative cluster control module 23 secures oneunused line of the control software management table 25, stores thecontrol software ID unique to that control software 20 which wasassigned to the control software 20 activated in step S56 in the controlsoftware ID column 25A (FIG. 6 ) of that line, stores the status of thatcontrol software 20 (“Active” or “Passive”) in the status column 25B(FIG. 6 ) of that line, stores the group ID of the redundancy group 26as the same as the inoperable control software 20 in the group ID column25C (FIG. 6 ) of that line, and stores the node ID of the deploymentdestination storage node 3 in the operating node ID column 25D (FIG. 6 )of that line.

Furthermore, the representative cluster control module 23 issues acommand to the cluster control module 23 of another storage node 3 inthe same cluster 6 so as to update the control software management table25 in the same manner as described above.

When the representative cluster control module 23 completes updating thecontrol software management table 25 of the respective storage nodes 3in the cluster 6 as described above, the representative cluster controlmodule 23 returns to step S50, and subsequently repeats step S50 onwardin the same manner.

The processing contents of the representative cluster control module 23in a case where the foregoing all control software migration command,which designates the storage nodes 3 to be removed, is issued are thesame as above. In the foregoing case, processing is executed uponexchanging the foregoing “inoperable control software 20” with “thecontrol software 20 running on the storage node 3 to be removed”.

(2-3-2) Deployment Destination Storage Node Determination Processing

FIG. 15 shows the specific processing routine of the deploymentdestination storage node determination processing to be executed by therepresentative cluster control module 23 in step S52 of the redundancyrecovery processing (FIG. 14 ).

When the representative cluster control module 23 proceeds to step S52of the redundancy recovery processing, the representative clustercontrol module 23 starts the deployment destination storage nodedetermination processing shown in FIG. 15 , and foremost refers to thestorage node management table 24 (FIG. 5 ), and then creates a list(this is hereinafter referred to as the “deployment destination nodecandidate list”) which arranges the node ID of each storage node 3 inthe cluster 6 in order from the storage node 3 having a low CPU load(S60). Here, when there are a plurality of storage nodes 3 that have thesame CPU load, the representative cluster control module 23 sets thestorage nodes 3 with fewer control software 20 in operation near the topof the list among the foregoing storage nodes 3.

Subsequently, the representative cluster control module 23 selects thetop node ID among the respective storage nodes 3 in which their node IDis included in the deployment destination node candidate list created instep S60 (S61), and determines, by referring to the control softwaremanagement table 25 (FIG. 6 ), whether or not control software 20 (thatis, intragroup control software 20) belonging to the same redundancygroup 26 as the inoperable control software 20 is running on the storagenode (this is hereinafter referred to as the “first selected storagenode”) 3 corresponding to the selected node ID (S62). The representativecluster control module 23 proceeds to step S65 upon obtaining a positiveresult in the foregoing determination.

Meanwhile, when the representative cluster control module 23 obtains anegative result in the determination of step S62, the representativecluster control module 23 refers to the storage node management table24, and determines whether or not the first selected storage node 3belongs to the same failure group as the storage node loaded withanother control software belonging to the same redundancy group 26 asthe inoperable control software 20 (S63). The representative clustercontrol module 23 proceeds to step S65 upon obtaining a positive resultin the foregoing determination.

Meanwhile, when the representative cluster control module 23 obtains anegative result in the determination of step S63, the representativecluster control module 23 refers to the storage node management table24, and determines whether or not the capacity, CPU load, memory amountand communication band of the backend network 5 of the storage device ofthe first selected storage node 3 will exceed the tolerance level whenthe control software 20 to become the substitute of the inoperablecontrol software 20 is deployed in the first selected storage node 3(S64). The used capacity, CPU load, memory used amount and usedcommunication band to be used by the substitute control software 20 canbe determined, for instance, by storing, from the control softwaremanagement table 25, the CPU load, memory used amount and usedcommunication band, which were being used by the inoperable controlsoftware 20 immediately after the occurrence of the failure.

When the representative cluster control module 23 obtains a negativeresult in the determination of step S64, the representative clustercontrol module 23 determines the storage node (first selected storagenode) 3, which was assigned the node ID selected in immediatelypreceding step S61, as the foregoing deployment destination storage node3 (S67), subsequently ends the deployment destination storage nodedetermination processing, and returns to the redundancy recoveryprocessing (FIG. 14 ).

Meanwhile, when the representative cluster control module 23 obtains apositive result in the determination of step S64, the representativecluster control module 23 deletes the node ID selected in immediatelypreceding step S61 from the deployment destination node candidate list(S65), and thereafter determines whether or not the deploymentdestination node candidate destination list is now empty (S66).

When the representative cluster control module 23 obtains a negativeresult in the foregoing determination, the representative clustercontrol module 23 returns to step S61, and subsequently executes theprocessing of step S61 onward in the same manner. Moreover, when therepresentative cluster control module 23 obtains a positive result inthe determination of step S66, the representative cluster control module23 ends the deployment destination storage node determinationprocessing.

(2-3-3) Rebalance Processing

Meanwhile, FIG. 16 shows the processing routine of the rebalanceprocessing to be periodically or randomly executed by the representativecluster control module 23 in the cluster when a storage node 3 has beenadded to the cluster 6, and even when a storage node 3 has not beenadded to the cluster 6. The representative cluster control module 23, inaccordance with this processing routine, distributes the load of therespective storage nodes 3 in the cluster 6 by migrating the function ofany one of the control software 20 in the cluster 6 to another storagenode 3 (when a storage node 3 has been added to the cluster 6, then theadded storage node (this is hereinafter referred to as the “addedstorage node”) 3) in that cluster 6.

In effect, when the representative cluster control module 23 recognizesthat a new storage node 3 has been added or that the periodical orrandom execution timing of the rebalance processing has arrived, therepresentative cluster control module 23 starts the rebalance processingshown in FIG. 16 , and foremost executes the migration target controlsoftware and migration destination storage node determination processingof determining the control software (this is hereinafter referred to asthe “migration target control software”) 20 to be migrated to anotherstorage node 3 (including the added storage node 3) in the cluster 6 andthe storage node (this is hereinafter referred to as the “migrationdestination storage node”) 3 to become the migration destination of themigration target control software 20, respectively (S70).

For example, when each redundancy group 26 is adopting an active-passiveconfiguration, the representative cluster control module 23 determinesone control software 20 set to the passive mode among the controlsoftware 20 loaded in a storage node having a high load as the migrationtarget control software 20. Moreover, when a storage node 3 is added tothe cluster 6, the representative cluster control module 23 determinesthe added storage node 3 as the migration destination storage node 3.

Subsequently, the representative cluster control module 23 notifies thecontrol software ID of the migration target control software 20 and thenode ID of the storage node (migration destination storage node) 3 tobecome the migration destination of the migration target controlsoftware 20 to the cluster control module 23 of the storage node 3 inwhich the migration target control software 20 determined in step S70 isrunning (S71).

Consequently, the cluster control module 23 that received the foregoingnotice issues a command to the redundancy module 22 in its own storagenode 3 so as to transfer the configuration information 21 retained inthe migration target control software 20 to the notified migrationdestination storage node 3. Moreover, the redundancy module 22 thatreceived the foregoing command transfers the configuration information21 retained in the migration target control software 20 to the migrationdestination storage node 3 via the backend network 5. When the clustercontrol module 23 of the storage node 3 receives the configurationinformation 21, the cluster control module 23 stores the receivedconfiguration information 21 in the memory 11. Moreover, the redundancymodule 22 subsequently transfers difference data to the migrationdestination storage node 3 via the backend network 5 so as tosynchronize the configuration information 21 retained in the migrationtarget control software 20, and the copy of the configurationinformation 21 retained in the migration destination storage node 3.

Meanwhile, after the processing of step S71, the representative clustercontrol module 23 waits for the synchronization of the configurationinformation 21 retained in the migration target control software 20 andthe configuration information 21 copied to the migration destinationstorage node 3 (S72). When the synchronization of the two configurationinformation 21 is eventually completed, the representative clustercontrol module 23 issues a command to the cluster control module 23 ofthe migration destination storage node 3 so as to activate the controlsoftware 20 to become the substitute of the migration target controlsoftware 20 (S73).

Consequently, the cluster control module 23 of the migration destinationstorage node 3 that received the foregoing command activates the controlsoftware 20 to become the substitute of the migration target controlsoftware 20. Here, the cluster control module 23 activates the controlsoftware 20 in the passive mode when the migration target controlsoftware 20 was operating in the passive mode, and activates the controlsoftware 20 in the active mode when the migration target controlsoftware 20 was operating in the active mode.

Subsequently, the representative cluster control module 23 issues acommand to the cluster control module 23 of the storage node 3, in whichthe migration target control software 20 is running, so as to stop theoperation of the migration target control software 20 (S74).Consequently, the cluster control module 23 that received the foregoingcommand stops the operation of the migration target control software 20.

Subsequently, the representative cluster control module 23 updates thecontrol software management table 25 (FIG. 6 ) retained in its ownstorage node 3 to a status after exchanging the migration target controlsoftware 20 and the control software 20 activated in step S73 asdescribed above, and issues a command to the other cluster controlmodule 23 to update the control software management table 25 (S75).

Specifically, the representative cluster control module 23 updates thenode ID stored in the operating node ID column 25D (FIG. 6 ) of the linecorresponding to the migration target control software 20 in the controlsoftware management table 25 in its own storage node 3 to the node ID ofthe migration destination storage node 3. Moreover, the representativecluster control module 23 issues a command to the cluster control module23 of the other storage nodes 3 in the same cluster 6 so as to updatethe control software management table 25 in the same manner.

Next, the representative cluster control module 23 generates a newredundancy group 26, and respectively determines the deploymentdestination storage nodes 3 of the respective control software 20configuring that redundancy group 26 (S76). In the foregoing case, therepresentative cluster control module 23 determines the deploymentdestination of one control software 20 among the respective controlsoftware 20 configuring the redundancy group 26 to be the migrationdestination storage node 3, and determines the deployment destination ofthe other control software 20 based on the deployment destinationstorage node determination processing described above with reference toFIG. 15 .

Furthermore, the representative cluster control module 23 deploys therespective control software 20 configuring the new redundancy group 26in the storage node 3 determined in step S76 according to thedetermination result of step S76 (S77). Specifically, the representativecluster control module 23 issues a command to the migration destinationstorage node 3 and the cluster control module 23 of the respectivestorage nodes 3 determined in step S76 so as to activate new controlsoftware 20. Consequently, each cluster control module 23 that receivedthe foregoing command activates unused control software 20 in its ownstorage node 3.

Here, the representative cluster control module 23 notifies the node IDof the storage node 3 loaded with the other control software 20 to thecluster control module 23 of the storage node 3 (migration destinationstorage node 3) loaded with the control software 20 to be set to theactive mode of the two control software 20 activated in step S77 (S78).

Consequently, the cluster control module 23 that received the foregoingnotice notifies the control software ID of the other control software 20and the node ID of the storage node 3 loaded with the other controlsoftware 20 to the redundancy module 22 in its own storage node 3.Moreover, the cluster control module 23 that received the foregoingnotice causes the control software 20 to start receiving read requestsor write requests, and, by sending, as difference data, the differencein comparison to the original configuration information 21 to thestorage node 3 loaded with the other control software 20 each time theconfiguration information 21 retained in that control software 20 isupdated, the configuration information 21 retained in the other controlsoftware 20 is updated in the same manner.

Subsequently, the representative cluster control module 23 updates thecontrol cluster management table 25 (FIG. 6 ) in its own storage node 3based on the processing result of step S76 to step S78, and additionallyupdates the control cluster management table 25 in the other storagenodes 3 in the same manner (S79).

Specifically, the representative cluster control module 23 newlyregisters the respective control software 20 configuring the newlycreated redundancy group 26 in the control cluster management table 25in its own storage node 3, and, stores “Active” in the status column 25B(FIG. 6 ) of the line corresponding to the control software 20 runningon the migration destination storage node 3 among the foregoing controlsoftware 20, and stores “Passive” in the status column 25B of the linecorresponding to the other control software 20 among the foregoingcontrol software 20. Moreover, the representative cluster control module23 stores the group ID, which was assigned to the newly createdredundancy group, in the group ID column 25C (FIG. 6 ) of that line, andadditionally stores the node ID of the storage nodes 3, on which thecorresponding control software 20 is running on the operating node IDcolumn (FIG. 6 ) of that line.

Furthermore, the representative cluster control module 23 notifies thecontents of the thus updated control software management table 25, whichit is managing, to the cluster control modules 23 of the respectivestorage nodes 3 in the same cluster 6. Consequently, each clustercontrol module 23 that received the foregoing notice updates the controlsoftware management table 25, which it is managing, to be the samecontents as the contents of the new control software management table 25notified from the representative cluster control module 23.

The representative cluster control module 23 thereafter ends therebalance processing.

(2-3-4) Migration Target Control Software and Migration DestinationStorage Node Determination Processing

FIG. 17 shows the specific processing routine of the migration targetcontrol software and migration destination storage node determinationprocessing to be executed by the representative cluster control module23 in step S70 of the rebalance processing.

When the representative cluster control module 23 proceeds to step S70of the rebalance processing, the representative cluster control module23 starts the migration target control software and migrationdestination storage node determination processing shown in FIG. 17 , andforemost refers to the storage node management table 24 (FIG. 5 ), andthen creates a list (this is hereinafter referred to as the “migrationsource node candidate list”) which arranges the node ID of each storagenode 3 in the cluster 6 in order from the storage node 3 having a lowCPU load (S80). Here, when there are a plurality of storage nodes 3 thathave the same CPU load, the representative cluster control module 23sets the storage nodes 3 with the fewer control software 20 in operationnear the top of the list among the foregoing storage nodes 3.

Subsequently, the representative cluster control module 23 selects thetop node ID among the respective storage nodes 3 in which their node IDis included in the migration source node candidate list created in stepS80 (S81), and determines, by referring to the control softwaremanagement table 25 (FIG. 6 ), whether or not one or more controlsoftware 20 are running on the storage node (this is hereinafterreferred to as the “second selected storage node”) 3 corresponding tothe selected node ID (S82). The representative cluster control module 23proceeds to step S86 upon obtaining a negative result in the foregoingdetermination.

Meanwhile, when the representative cluster control module 23 obtains apositive result in the determination of step S82, the representativecluster control module 23 selects one control software 20 among thecontrol software 20 running on the storage node 3 of the node IDselected in step S81, and determines the selected control software 20 asthe migration target control software 20 (S83). As the method ofselecting the migration target control software 20, for instance,control software 20 set to the passive mode is selected in order tosuppress influence on the I/O performance during migration. Otherwise,when there is no control software 20 set to the passive mode, controlsoftware 20 having a low CPU load is selected among the control software20 set to the active mode.

Subsequently, the representative cluster control module 23 determinesthe deployment destination storage node (migration destination storagenode) 3 of the migration target control software 20 by executing thedeployment destination storage node determination processing describedabove with reference to FIG. 15 (S84).

Furthermore, the representative cluster control module 23 whether or notit was possible to determine the deployment destination storage node(migration destination storage node) 3 of the migration target controlsoftware 20 based on the deployment destination storage nodedetermination processing of step S84 (S85). When the representativecluster control module 23 obtains a positive result in the foregoingdetermination, the representative cluster control module 23 ends themigration target control software determination processing and returnsto the rebalance processing.

Meanwhile, when the representative cluster control module 23 obtains anegative result in the determination of step S85, the representativecluster control module 23 deletes the node ID selected in step S81 fromthe migration source node candidate list (S86), and thereafterdetermines whether or not the migration source node candidate list isnow empty (S87).

When the representative cluster control module 23 obtains a negativeresult in the foregoing determination, the representative clustercontrol module 23 returns to step S81, and subsequently executes theprocessing of step S81 onward in the same manner. Moreover, when therepresentative cluster control module 23 obtains a positive result inthe determination of step S87, the representative cluster control module23 ends the migration target control software determination processingand returns to the rebalance processing.

(3) Effect of this Embodiment

As described above, with this embodiment, because the control software20 configuring the same redundancy group 26 are deployed by beingdistributed to different storage nodes 3 so that the load of therespective storage nodes 3 configuring the cluster 6 is distributed, thecontrol software 20 are distributed and deployed in the respectivestorage nodes 3.

Thus, according to the information processing system 1 of thisembodiment, the processing load can be distributed in the respectivenodes 3 in the cluster 6, and information processing resources such asthe CPU 10, the memory 11, the storage device 12 and the first andsecond communication devices 13, 14 of the respective storage nodes 3can be effectively utilized.

Moreover, with the information processing system 1, even when any one ofthe control software 20 of the redundancy group 26 becomes inoperabledue to the removal of the storage node 3 or a failure in the storagenode 3, because the substitute control software 20 or the configurationinformation 21 to be used by that control software 20 is immediatelyreproduced in the appropriate storage node 3, the redundantconfiguration of the control software 20 can be maintained constantlyand reliably, and the availability and reliability as the overall systemcan be improved by that much.

Furthermore, with the information processing system 1, because any oneof the control software 20 in the cluster is redeployed, upon adding astorage node 3, in that storage node 3, the load of the storage nodescan be distributed without having to trouble the administrator of theinformation processing system 1, and the management as the overallsystem can be facilitated by that much.

(4) Other Embodiments

Note that, while the foregoing embodiments explained a case of therepresentative cluster control module 23 executing the processing ofFIG. 14 to FIG. 17 , the present invention is not limited thereto, andit is also possible to connect a management device (management unit),which is configured from a computer device such as a server device, tothe storage service network 4 and the backend network 5 separately fromthe storage nodes 3, and the management device may execute a part or allof the processing of FIG. 14 to FIG. 17 in substitute for therepresentative cluster control module 23.

Moreover, while the foregoing embodiments explained a case of executingthe rebalance processing described above with reference to FIG. 16 whena storage node 3 was added to the cluster 6, the present invention isnot limited thereto, and rebalance processing of redeploying necessarycontrol software 20 in other storage nodes 3 so as to distribute theload of the respective storage nodes 3 may be performed as needed.

Moreover, while the foregoing embodiments explained a case of executingthe rebalance processing (FIG. 16 ) of deploying the control software 20so as to distribute the load of all storage nodes 3, the presentinvention is not limited thereto, and it is also possible to adopt adistribution method of not deploying the control software 20 in theactive mode, or not deploying the control software 20 irrespective ofwhether it is in the active mode or the passive mode, in certain storagenodes 3 during a normal period. In other words, by preparing storagenodes 3 with sufficient capacity, CPU load, memory amount, andcommunication band of the backend network 5 of the storage device, it ispossible to a distribution method capable of preventing the shortage ofcalculation resources when a failure occurs.

Moreover, a hypervisor may run on a server, one or more virtual machinesmay run on the hypervisor, and the various programs shown in FIG. 3 mayrun on the virtual machines. In other words, the various programs(control software 20, redundancy module 22, cluster control module 23)may run on the hardware of a physical computer, or run on a virtualmachine. Similarly, the compute node 2 may be an application program(host program) that runs on a virtual machine, or a physical hostcomputer (host computer). When the information processing system 1 has aplurality of servers, certain servers may be installed at differentsites. Moreover, a part or all of the servers of the informationprocessing system 1 may be stored in a cloud, and services may beprovided to users via a network.

The present invention may be a configuration (hyper convergedinfrastructure) in which a virtual machine on which various programs(control software 20, redundancy module 22, cluster control module 23)are operating and a virtual machine on which the host program isoperating are on the same server (node), or a configuration in whichthey are on different servers which are connected via a network.

INDUSTRIAL APPLICABILITY

The present invention can be broadly applied to information processingsystems of various configurations comprising a plurality of storagenodes.

REFERENCE SIGNS LIST

1 . . . information processing system, 2 . . . compute node, 3 . . .storage node, 4 . . . storage service network, 5 . . . backend network,6 . . . cluster, 10 . . . CPU, 11 . . . memory, 12 . . . storage device,20 . . . control software, 21 . . . configuration information, 22 . . .redundancy module, 23 . . . cluster control module, 24 . . . storagenode management table, 25 . . . control software management table.

The invention claimed is:
 1. A storage system including a plurality ofstorage nodes and one or more storage devices which respectively providea storage area, wherein the storage system comprising: one or morecontrol software which read/write requested data from/into acorresponding storage device according to a request from a higher-leveldevice; and a management unit that manages each of the control softwarein a cluster configured from the plurality of storage nodes, whereineach of the control software retains predetermined configurationinformation required for reading/writing requested data from/into thecorresponding storage device according to a request from thehigher-level device, wherein a plurality of the control software aremanaged as a redundancy group, and the predetermined configurationinformation retained in each of the control software belonging to a sameredundancy group is synchronously updated, wherein control softwaremanagement information that stores a status of each of the controlsoftware, a redundancy group to which each of the control softwarebelongs, and volumes of which the redundancy group is in charge, isretained, wherein when the management unit receives a request from thehigher-level device, the management unit identifies a storage node, inwhich the plurality of the control software for a redundancy group incharge of a volume related to the request is deployed, on a basis of thecontrol software management information and transfers the request fromthe higher-level device to the identified storage node, and wherein whena storage node is added to the storage system, the management unitmigrates control software which is running on an existing storage nodeto the added storage node, configures a new redundancy group which has aplurality of control software, and deploys the plurality of controlsoftware of the new redundancy group to a plurality of storage nodesincluding the added storage node.
 2. The storage system according toclaim 1, wherein each of the storage nodes has the control softwaremanagement information, and wherein when the control software managementinformation is updated for one node, the updated control softwaremanagement information is reflected in the control software managementinformation of other storage nodes.
 3. The storage system according toclaim 1, wherein at least one of the one or more control software amongthe plurality of control software configuring the same redundancy groupis set to a first status of receiving a request from the higher-leveldevice, and the remaining one or more control software belonging to acorresponding redundancy group are set to a second status of notreceiving a request from the higher-level device, wherein the pluralityof control software belonging to different redundancy groups aredeployed in a same storage node, and wherein the management unittransfers the request from the higher-level device to a storage nodewhere the at least one of the one or more control software in the firststatus of the redundancy group in charge of the volume related to therequest is deployed.
 4. The storage system according to claim 3,wherein, upon removing the storage node or when the storage node issubject to a failure, the management unit switches, to the first status,any one of the one or more control software in the second statusbelonging to the same redundancy group as the one or more controlsoftware in the first status deployed in a corresponding storage node.5. The storage system according to claim 4, wherein the control softwaremanagement information has a storage node resource usage amount of theone or more control software, and wherein, when there are a plurality ofcandidates of the one or more control software to succeed processing ofthe one or more control software set to the first status, the managementunit causes, among the plurality of candidates of the one or morecontrol software, the one or more control software deployed in thestorage node with a low load, to succeed the processing on the basis ofthe control software management information.
 6. The storage systemaccording to claim 1, wherein, with respect to the redundancy group inwhich a number of control software has decreased due to a reduction orfailure of the storage node, the management unit activates new controlsoftware to substitute the one or more control software which wasdeployed in the storage node that was reduced or subject to a failure inthe storage node in which the one or more control software belonging tothe corresponding redundancy group has not been deployed.
 7. The storagesystem according to claim 6, wherein the control software managementinformation has a storage node resource usage amount of the one or morecontrol software, and wherein, when there are a plurality of storagenodes which may become a deployment destination of the one or morecontrol software which is new to become a substitute for the one or morecontrol software which was deployed in the storage node, the managementunit causes the storage node with a low, to become the deploymentdestination of the new control software to become the substitute for theone or more control software, on the basis of the control softwaremanagement information.
 8. The storage system according to claim 1,wherein, when the storage node is added in the cluster, the managementunit redeploys certain control software, which were deployed in otherstorage nodes in the cluster, to the added storage node.
 9. The storagesystem according to claim 1, wherein, when the storage node is added inthe cluster, the management unit generates a new redundancy group,deploys one of the one or more control software, which belongs to thenew redundancy group, to the added storage node, and deploys othercontrol software, which belong to the new redundancy group, to otherstorage nodes.
 10. The storage system according to claim 9, wherein thecontrol software management information has a storage node resourceusage amount of the one or more control software, and wherein, whenthere are a plurality of storage nodes which may become a deploymentdestination of the other control software belonging to the redundancygroup which is new, the management unit causes the storage node with alow load, to become the deployment destination of the other controlsoftware belonging to the new redundancy group on the basis of thecontrol software management information.
 11. The storage systemaccording to claim 1, wherein a node to deploy the one or more controlsoftware of the new redundancy group includes a storage node at whichthe one or more control software migrated to the added storage node wasdeployed before being migrated.
 12. A control software deployment methodof deploying, in a storage system including a plurality of storage nodeseach equipped with one or more storage devices which respectivelyprovide a storage area, one or more control software which read/writerequested data from/into a corresponding storage device according to arequest from a higher-level device, and a management unit that manageseach of the one or more control software in a cluster configured fromthe plurality of storage nodes, wherein each of the one or more controlsoftware retains predetermined configuration information required forreading/writing requested data from/into the corresponding storagedevice according to a request from the higher-level device, and whereinthe one or more control software deployment method comprises the stepsof: managing a plurality of the one or more control software as aredundancy group, and synchronously updating the predeterminedconfiguration information retained in each of the one or more controlsoftware belonging to a same redundancy group; retaining controlsoftware management information that stores a status of the one or morecontrol software, a redundancy group to which the one or more controlsoftware belongs, and volumes of which the redundancy group is incharge; when the management unit receives a request from thehigher-level device, identifying, by the management unit, a storagenode, in which the one or more control software for a redundancy groupin charge of a volume related to the request is deployed, on a basis ofthe control software management information and transferring, by themanagement unit, the request from the higher-level device to theidentified storage node; and when a storage node is added to the storagesystem, migrating, by the management unit, the one or more controlsoftware which is running on an existing storage node to the addedstorage node, configuring, by the management unit, a new redundancygroup which has a plurality of control software, and deploying, by themanagement unit, the plurality of control software of the new redundancygroup a plurality of storage nodes including the added storage node.